Imaging device and image processing method

ABSTRACT

There is provided an imaging device including: an image processing unit; a face recognition processing unit; a storage unit; and a composition processing unit for generating composite data by a composition process so that persons photographed in each of a plurality of image data are included in one image data. The face recognition processing unit recognizes a first person by performing a face recognition process on first image data. When second image data obtained by photographing a second person at the different photographing timing of the first image data, with the same background as the first image data, is recorded in the storage unit, the composition processing unit generates composite data in which the first person and the second person are superimposed on the same background.

CROSS REFERENCE

This application is a Continuation of U.S. patent application Ser. No.16/709,476, filed on Dec. 10, 2019, which is a continuation of Ser. No.15/739,097, filed on Dec. 21, 2017, now U.S. Pat. No. 10,542,224, issuedJan. 21, 2020, which is the U.S. National Phase under 35 U.S.C. § 371 ofInternational Application No. PCT/JP2015/068535, filed on Jun. 26, 2015,the entire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to an imaging device and an imageprocessing method. More particularly, the present invention relates to atechnique for compositing image data obtained by photographing aplurality of persons.

BACKGROUND ART

As background art of this technical field, Patent Literature 1 addressesthe problem of “being able to easily take a group photo of a pluralityof persons including the photographer”, and discloses the followingtechnique as a means for solving this problem. “In a digital camera, aphotographed image “a” of (a plurality of) persons to be photographedother than a photographer is photographed by the photographer, justafter a composition imaging mode is instructed. Next, the digital camerais given to any one of the other persons to be photographed and aphotographed image “b” of only the former photographer is photographed.Then, in a predetermined composition condition, for example, when aplurality of faces with a size greater than a predetermined size arepresent in one photographed image and one face with a size greater thanthe predetermined size is present in the other photographed image, orwhen the composition of persons with the faces detected in each of thetwo photographed images a and b is different, a new composite image “c”is generated by compositing a face image included in either one of thetwo photographed images “a” and “b” that are continuously photographed,with the other image. This facilitates the composition of a group photoin which all the plurality of persons are photographed.”

CITATION LIST Patent Literature

Patent Literature 1: US Patent No. 2012/120273 (extract)

SUMMARY OF INVENTION Technical Problem

In the above described Patent Literature 1, when the photographed images“a” and “b” are continuously photographed (hereinafter referred to as “aseries of photographing operations”) just after a composition imagingmode is instructed, a composite image “c” is generated based on thesephotographed images. However, for example, when another shutter chancecomes during a series of photographing operations, another image may notbe photographed until the series of photographing operations iscompleted, so that the shutter chance may be missed. In addition, if onepicture is not acceptable, it is necessary to re-photograph it from thebeginning. Thus, there is room for further improvement in operabilityfor generating a composite photo.

The present invention has been made in view of these circumstances, andan object of the present invention is to provide an imaging device withimproved operability for easily generating a photo in which all theplurality of persons are included, as well as an image processingmethod.

Solution to Problem

The technology described in the claims is used as a means for solvingthe above described problems. To cite an example, there is provided animaging device including: an image processing unit for photographing anobject to generate image data; a face recognition processing unit forperforming a face recognition process on the image data; an imagerecording unit for recording the image data; and a compositionprocessing unit for generating a composite data by performing acomposition process so that the photographed persons in each of aplurality of image data are included in one image data. The facerecognition processing unit recognizes the face of a first person byperforming the face recognition process on first image data. When secondimage data obtained by photographing a second person other than thefirst person at an arbitrary photographing timing, which is differentfrom the photographing timing of the first image data, with the samebackground as the background of the first image data, is recorded in theimage recording unit, the composition processing unit generates thecomposite data in which the first person and the second person aresuperimposed on the same background by using the first image data andthe second image data.

Advantageous Effects of Invention

According to the present invention, it is possible to provide an imagingdevice with improved operability for easily generating a photo in whichall the plurality of persons are included, as well as an imageprocessing method. Problems, configurations, and effects other thanthose described above will become apparent based on the followingdescription of embodiments of the present invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a block diagram showing a configuration example of an imagingdevice according to a first embodiment.

FIG. 1B is a diagram showing the software configuration of the imagingdevice according to the first embodiment.

FIG. 2A is a front view of the imaging device according to the firstembodiment.

FIG. 2B is a back view of the imaging device according to the firstembodiment.

FIG. 3 is a flow chart showing an entire operation example of theimaging device according to the first embodiment.

FIG. 4 is a flow chart showing another operation example (automaticcomposition mode) of the imaging device according to the firstembodiment.

FIG. 5 is a flow chart showing an operation example of a compositionprocessing unit according to the first embodiment.

FIG. 6A is a view showing a display example of a display unit just afterphotographing by the imaging device according to the first embodiment.

FIG. 6B is a view showing a display example of the image data composedby the imaging device according to the first embodiment.

FIG. 7A is a view showing a display example of a selection screen forselecting whether or not to record the image data composed by theimaging device according to the first embodiment.

FIG. 7B is a view showing a display example when the image is notcomposed by the imaging device according to the first embodiment.

FIG. 8 is a view showing a display example of a selection screen forselecting whether or to display the image data composed with anotherimage by the imaging device according to the first embodiment.

FIG. 9 is an illustrative example of the search and composition of imagedata according to the first embodiment.

FIG. 10 is an illustrative example of the search and composition ofimage data according to the first embodiment.

FIG. 11 is a flow chart showing an entire operation example of theimaging device according to a second embodiment.

FIG. 12 is a flow chart showing an operation example of a compositionprocessing unit according to the second embodiment.

FIG. 13A is a view showing a display example of the display unit when areproduction button is pressed in the imaging device according to thesecond embodiment.

FIG. 13B is a view showing a display example of the display unit whenthe image data is changed to another image data in the imaging deviceaccording to the second embodiment.

FIG. 14A is a view showing a display example of the display unit when acomposition button is pressed in the imaging device according to thesecond embodiment.

FIG. 14B is a view showing a display example of the display unit whenthe image data is changed to another image data in the imaging deviceaccording to the second embodiment.

FIG. 15 is an illustrative example of the search and composition ofimage data according to the second embodiment.

FIG. 16 is a flow chart showing an entire operation example of an imagedevice according to a third embodiment.

FIG. 17 is a flow chart showing an operation example of a compositionprocessing unit according to the third embodiment.

FIG. 18A is a view showing a display example of the display unit justafter photographing or when the reproduction button is pressed in theimaging device according to the third embodiment.

FIG. 18B is a view showing a display example of the image data composedby the imaging device according to the third embodiment.

FIG. 19 is an illustrative example of the search and composition ofimage data according to the third embodiment.

FIG. 20A is a block diagram showing a configuration example of theimaging device according to a fourth embodiment.

FIG. 20B is a diagram showing the software configuration of the imagingdevice according to the fourth embodiment.

FIG. 21 is a flow chart showing an operation example of a compositionprocessing unit according to the fourth embodiment.

FIG. 22 is an illustrative example of the search and composition ofimage data according to the fourth embodiment.

DESCRIPTION OF EMBODIMENT

Hereinafter, preferred embodiments of the present invention will bedescribed with reference to the accompanying drawings. Throughout thedrawings, the same components are denoted by the same reference numeralsand the repetitive description thereof will be omitted. The followingembodiments will be described by an example of an imaging deviceaccording to the present invention.

First Embodiment

A first embodiment is to compose the image data on display and the imagedata selected by an imaging device are composed in manual compositionmode (operation mode for composition after composition instruction bythe user) or in automatic composition mode (operation mode for automaticcomposition by the imaging device).

[Hardware Configuration of the Imaging Device]

FIG. 1A is a block diagram showing a configuration example of an imagingdevice according to the first embodiment of the present invention.

In FIG. 1A, a main controller 101 configured with a CPU (CentralProcessing Unit), or the like, controls the whole of an imaging device 1according to various operation programs and data stored in ROM (ReadOnly Memory) 102 or RAM (Random Access Memory) 103. A system bus 100 isa data communication path for transmitting and receiving data betweenthe controller 101 and each of the units within the imaging device 1.

The ROM 102 is a memory in which various programs are stored to controlthe imaging device 1. For example, rewritable ROM, such as EEPROM(Electrically Erasable Programmable ROM) or flash ROM, is used.

The RAM 103 is used as a temporary storage area when executing theprograms stored in the ROM 102, or as a temporary storage of thephotographed images.

A storage unit 104 (corresponding to the image recording unit) is tostore information such as operation setting values and the like of theimaging device 1. For example, a nonvolatile rewritable device such as aflash ROM or SSD (Solid Sate Drive) is used.

The ROM 102 and the RAM 103 may be integrated into the main controller101. Further, a part of the storage area within storage unit 104 may beused for the ROM 102, instead of configuring it as an independent unitas shown in FIG. 1A.

An external recording medium interface (I/F) 105 is an interface forrecording and reading information from an external recording medium 106,such as SD card, which is attachable to and detachable from the imagingdevice 1 like an SD card.

An external interface 107 is a group of interfaces to enhance thefunction of the imaging device 1. In the present embodiment, theexternal interface 107 includes a USB (Universal Serial Bus) interface(I/F) 107 a, a video output interface (I/F) 107 b, and an audio outputinterface (I/F) 107 c. The USB interface 107 a is connected to the USBinterface of an external device, such as a personal computer or atelevision receiving device, to allow reading, display, or the like ofimage data stored in the storage unit 104 or the external recordingmedium 106. The video output interface 107 b and the audio outputinterface 107 c output a video signal/audio signal to an externalvideo/audio output device. Here, the video output interface 107 b andthe audio output interface 107 c can also output video and audiotogether by using HDMI (High-Definition Multimedia Interface: registeredtrademark).

An audio input unit 108 is configured with a microphone for convertingthe peripheral sound of the imaging device 1 into an electrical signal,an A/D converter for converting the sound converted to the electricalsignal into digital signal audio data, and the like.

An audio signal processing unit 109 performs a filtering process on theaudio data input from the audio input unit, and other processes such asa process for converting the audio data according to the format of themoving image to be recorded in the external recording medium 106.

An imaging unit 110 is configured with the following devices: an opticalsystem device including a lens for zoom and focus operation; amechanical system device that drives the lens for zoom and focusoperation; and an electrical system device such as an image sensor suchas CCD or CMOS that converts an optical image of an object input fromthe lens into an electrical signal, and an A/D converter that convertsthe image converted to the electrical signal into a digital signal imagedata.

An image signal processing unit 111 performs image processing, such aswhite balance adjustment, exposure adjustment, and gamma correction, onthe image data input from the imaging unit 110. Further, the imagesignal processing unit 111 performs processes, such as, for example,converting a moving image to image data with a format such as MPEG2 orH.264, and converting a static image to image data with a format such asJPEG or TIFF, according to the format to be recorded in the externalrecording medium 106. Further, the image signal processing unit 111 alsoperforms composition process, face recognition process, pattern matchingprocess, and the like, on the image data stored in the storage unit 104or the external recording medium 106.

A display unit 112 displays an image of an object photographed by theimaging sensor of the imaging unit 110, an image recorded in the storageunit 104 or the external recording medium 106, or a screen forperforming various settings of the imaging device 1.

An operation unit 113 is an instruction input unit for inputting anoperation instruction to the imaging device 1, including a power ON/OFFbutton, a shutter button, a reproduction button, a composition button,and other buttons for performing various settings. Further, it may alsobe possible that a touch panel is provided on the front surface of thedisplay unit 112 so as to input an operation instruction to the imagingdevice 1 by detecting the position of the button or the like displayedon the display unit 112, as well as the position touched on the touchpanel.

A timer unit 114 measures the elapsed time, for example, from the dateand time set by the user, by using an RTC (Real Time clock) circuit, andoutputs date and time information. The date and time information isadded to the image data and recorded based on the date and timeinformation obtained from the timer unit 114.

A GPS receiver 115, which receives radio waves from a plurality of GPSsatellites, can obtain location (latitude, longitude, and the like)information based on the received signal. The location information isalso added to the image data and recorded.

A power supply unit 116 includes a battery (not shown). In response toan instruction from the main controller 101, the power supply unit 116supplies power to the respective units configuring the imaging device 1according to the power state such as power-on, power-off, or power-offstandby state.

The imaging device 1 may be a device such as digital camera, mobilephone, smartphone, tablet terminal, or navigation device as long as itincludes the imaging unit 110, or may be a PDA (Personal DigitalAssistants), or a notebook PC (Personal Computer). Further, the imagingdevice 1 may be a device having a communication function, such as amusic player or handheld game console, or other portable digitaldevices. Further, the imaging device 1 may also be a wearable smartwatch or smart glasses.

[Software Configuration of the Imaging Device]

FIG. 1B is a diagram showing the software configuration of the imagingdevice according to the first embodiment, showing the configuration ofthe software in the ROM 102, the RAM 103, and the storage unit 104.

In FIG. 1B, the main controller 101 executes the programs stored in theROM 102 to implement a basic operation execution unit 1021, a locationinformation acquisition unit 1022, an image processing unit 1023, animage recording unit 1024, an image reproduction unit 1025, a powermanagement unit 1026, a composition processing unit 1027, and a facerecognition processing unit 1028. Further, it is assumed that the RAM103 includes a temporary storage area 1031 for temporarily store data asappropriate in the execution of the program stored in the ROM 102, andan image temporary storage area 1032 in which the image processing unit1023 temporarily stores images photographed through the imaging unit110.

The storage unit 104 also includes areas, such as a setting informationstorage area 1041 for storing setting information for the execution ofthe program stored in the ROM 102, and an image storage area 1042 forstoring static image data and moving image data that are photographed bythe imaging device 1.

Note that in the following, in order to simplify the description, it isassumed that the main controller 101 executes the program stored in theROM 102 to control each operation block.

The basic operation execution unit 1021 controls various settings of theimaging device 1 as well as the entire operation.

The location information acquisition unit 1022 performs a process forobtaining the location information, such as latitude and longitude ofthe imaging device 1, based on the signals received by the GPS receiver115 from the GPS satellites. Further, the location informationacquisition unit 1022 may also have a Wi-Fi (registered trademark)reception function to increase the accuracy by correcting the locationinformation by using the information of the Wi-Fi radio wave receptionlevel, or obtain the location information by using the information ofthe Wi-Fi radio reception level.

When the shutter button included in the operation unit 113 is pressed,the image processing unit 1023 performs the process of importing imagedata of an object photographed by the imaging unit 110 into the imagetemporary storage area 1032 of the RAM 103.

The image recording unit 1024 performs the process of recording theimage data imported in the image temporary storage area 1032 into theimage storage area 1042 of the storage unit 104, or recording the imagedata into the external recording medium 106 through the externalrecording medium interface 105.

The image reproduction unit 1025 performs the process of displaying theimage of the object, which is photographed by the image sensor of theimaging unit 110, on the display unit 112 through the image temporarystorage area 1032 of the RAM 103. Further, the image reproduction unit1025 performs the process of reading the image data recorded in theexternal recording medium 106 to display on the display unit 112.

The power management unit 1026 performs the process of controlling thepower to be supplied to the respective units configuring the imagingdevice 1 with a power button included in the operation unit 113.

When the composition button included in the operation unit 113 ispressed or when display of composed image data is selected, thecomposition processing unit 1027 performs the process of compositing theimage data stored in the storage unit 104 or the external recordingmedium 106.

The face recognition processing unit 1028 performs a face recognitionprocess on the image data stored in the RAM 103, the storage unit 104 orthe external recording medium 106.

The programs stored in the ROM 102 as described above may be stored inadvance at the time of product delivery. Alternatively, it may also bepossible to store programs obtained from a server device or the like onthe Internet by a personal computer, through the USB interface 107 a.

Note that a buffer memory for temporarily storing image data may beprovided in the imaging unit 110 or the image signal processing unit111, instead of the image temporary storage area 1032 provided in theRAM 103.

[External View of the Imaging Device]

The exterior of the imaging device according to the first embodimentwill be described with reference to FIGS. 2A and 2B.

FIG. 2A is a front view of the imaging device 1. A lens configuring theimaging unit 110 is provided in front of the imaging device 1. Further,a power button 113-1 and a shutter button 113-2, which configure theoperation unit 113, are provided in the upper part of the imaging device1.

FIG. 2B is a back view of the imaging device 1. In the back surface ofthe imaging device 1, there are provided, in addition to the displayunit 112, a select/determine operation button 113-3 configuring theoperation unit 113, a menu button 113-4 for performing various settingsand processes such as file operation process selection, a reproductionbutton 113-5 for reproducing the image data recorded in the storage unit104 or the external recording medium 106, and a composition button 113-6(corresponding to the composition instruction operation unit) forcompositing the image data. A touch sensor may be provided in thedisplay unit 112 so that the user can operate the imaging device 1through the touch sensor, instead of pressing down the shutter button113-2, the select/determine operation button 113-3, the menu button113-4, the reproduction button 113-5, and the composition button 113-6.

[Entire Operation of the Imaging Device]

FIG. 3 is a flow chart showing an entire operation example of theimaging device according to the first embodiment. FIG. 3 describes theflow of operation when the image composition process (S313) is performedin a manual mode, in other words, triggered by the input operationperformed by the user to instruct image composition, and morespecifically by a “press of the image composition button (S312: YES)” inthe following example.

First, the power management unit 1026 confirms the press of the powerbutton 113-1. If the power button 113-1 is not pressed (S301: NO), thepower management unit 1026 stands by and waits in this step withouttransition to the next step.

If the power button 113-1 is pressed (S301: YES), the power managementunit 1026 performs the start-up process of the imaging device 1 (S302).When starting up, the imaging device 1 displays an image of an object,which is being photographed by the image sensor of the imaging unit 110,on the display unit 112.

The power management unit 1026 confirms the press of the power button113-1. If the power button 113-1 is pressed (S303: YES), the powermanagement unit 1026 performs the end process of the imaging device 1(S304).

If the power button 113-1 is not pressed (S303: NO), the imageprocessing unit 1023 confirms the press of the shutter button 113-2. Ifthe shutter button 113-2 is pressed (S305: YES), the imaging unit 110photographs the object to obtain image data (S306), and stores theobtained image data in the image temporary storage area 1032 of the RAM103. At this time, additional information, such as the type of device bywhich the image was photographed, aperture value, number of pixels, ISOsensitivity, photographing date and time, and location information, isalso stored into the image temporary storage area 1032. Then, the imagedata and the additional information that are stored in the imagetemporary storage area 1032 are stored in the storage unit 104 or theexternal recording medium 106 by the image recording unit 1024 (S307).Then, the image data, which is photographed by the imaging unit 110 andstored in the image temporary storage area 1032, is displayed on thedisplay unit 112 (S309).

If the shutter button 113-2 is not pressed (S305: NO), the imagereproduction unit 1025 confirms the press of the reproduction button113-5. If the reproduction button 113-5 is pressed (S308: YES), theimage data recorded in the storage unit 104 or the external recordingmedium 106 is reproduced by the image reproduction unit 1025 and thendisplayed on the display unit 112 (S309). The user can display the imageto be reproduced and displayed by pressing the reproduction button113-5. This image data displayed on the display unit 112, as well as theimage data generated by photographing the object by the user anddisplayed on the display unit 112 (S306) correspond to first image data.

FIG. 6A is a view showing a display example of the display unit 112 justafter photographing by the imaging device 1. A person A is photographedwith a mountain in the background. The display time of the photographedimage data is, for example, about 3 seconds, which may be changed by theuser. When a predetermined time interval has elapsed, the display of theimage data disappears. However, the image data is displayed again whenthe reproduction button 113-5 is pressed.

If the reproduction button 113-5 is not pressed (S308: NO), thecomposition processing unit 1027 confirms the press of the compositionbutton 113-6. If the composition button 113-6 is pressed (S312: YES),the image data obtained by photographing the object by the imaging unit110 and then recorded in the storage unit 104 or the external recordingmedium 106 through the image temporary storage area 1032 is composed bythe composition processing unit 1027 (S313). Then, the composed imagedata is displayed on the display unit 112 (S315).

FIG. 6B is a view showing a display example of the image data composedby the imaging device 1. The image including the person A photographedwith the mountain in the background is composed with a person Bphotographed with the same mountain in the background.

When selection and determination to record the composed image data intothe storage unit 104 or the external recording medium 106 is made by theselect/determine operation button 113-3 (S317: YES), the image recordingunit 1024 records the image data composed by the composition processingunit 1027 into the storage unit 104 or the external recording medium 106(S318).

FIG. 7A is a view showing a display example of the selection screen forselecting whether or not to record the image data composed by theimaging device 1. The user selects and determines “YES” or “NO” by theselect/determine operation button 113-3.

If selection and determination to avoid recording the composite imagedata in the storage unit 104 or the external recording medium 106 ismade by the select/determine operation button 113-3 (S317: NO) or if thecomposition button is not pressed (S312: NO), the operation returns toStep S303 after the image reproduction process (S315), or after theimage reproduction process (S309).

FIG. 7B is a view showing a display example when the image is notcomposed. If the image is not composed (if the composition failed as aresult of the image composition process in S313), the image shown inFIG. 7B is displayed on the display unit 112, instead of the image shownin FIG. 7A.

By performing the above described process from steps S301 to S318, theimaging device 1 records, reproduces, and composes the images.

Note that it may be possible to select the composition process by themenu button 113-4 and the select/determine operation button 113-3,instead of using the independent composition button 113-6, whenperforming the composition process. Further, it may also be possible toregister an important person in advance to automatically perform thecomposition operation when the registered person is photographed.Further, when the user takes a picture by touching an object displayedon the display unit 112 with a touch sensor, instead of pressing downthe shutter button 113-2, and if the object is a person, the particularperson may also be registered as an important person. Furthermore, whenthe photographed person is displaced from the center of the image by apredetermined value or more, or by a predetermined ratio or more, forexample, when the photographed person is displaced from the center by20% or more, the imaging device 1 may automatically perform thecomposition operation.

Next, with reference to FIG. 4, a description will be given of the imagecomposition process in a so-called automatic composition mode toautomatically perform the composition process when a person isphotographed. FIG. 4 is a flow chart showing another operation example(automatic composition mode) of the imaging device 1 according to thefirst embodiment.

Of the steps, S401 to S409 in FIG. 4 are the same as S301 to S309 inFIG. 3, so that the description thereof is omitted.

The composition processing unit 1027 confirms whether or not theautomatic composition mode is set (S412). The automatic composition modeis set in advance by the menu button 113-4 and the select/determineoperation button 113-3. For this reason, the menu button 113-4 and theselect/determine operation button 113-3 correspond to the mode settingunit.

In the case of the automatic composition mode (S412: YES), thecomposition processing unit 1027 composes the image data that isphotographed by the imaging unit 110 and then recorded in the storageunit 104 or the external recording medium 106 through the imagetemporary storage area 1032 (S413). The composition processing unit 1027performs the composition process in the image temporary storage area1032.

The display unit 112 displays a screen for selecting the presence orabsence of the display of the composed image data. The user makes aselection and determination of whether the composed image data isdisplayed, by the select/determine operation button 113-3 (S414: YES).

FIG. 8 is a view showing a display example of the selection screen forselecting whether the image data composed with another image by theimaging device 1 is displayed or not. The user selects and determines“YES” or “NO” with the select/determine operation button 113-3. Ifunable to compose the image data with another image data, the displayunit 112 does not display the selection screen.

Because S415 to 418 are the same as S315 to S318 in FIG. 3, thedescription thereof is omitted.

If the automatic composition mode is not set (S412: NO), or if thecomposed image data is not displayed (S414: NO), the operation returnsto S403.

By performing the above described process from S401 to S418, the imagingdevice 1 records, reproduces, and composes the images in the automaticcomposition mode.

[Image Composition Process]

Next, the image composition process in S313 and S411 will be described.

FIG. 5 is a flow chart showing an operation example of the compositionprocessing unit 1027.

The composition processing unit 1027 searches for the latest image withthe same photographing date as that of the image displayed on thedisplay unit 112, among the image data stored in the image storage area1042 of the storage unit 104 or outside the external recording medium106 (S501). Hereinafter, the searched image is referred to as the“search image”, and the image displayed on the display unit 112 isreferred to as the “display image”.

The composition processing unit 1027 confirms whether or not thephotographing location of the search image is the same as thephotographing location of the display image based on the locationinformation obtained by the location information acquisition unit 1022(S502).

If the photographing locations are the same (S502: YES), the compositionprocessing unit 1027 confirms whether or not the photographing date andtime of the search image and the photographing date and time of thedisplay image are within a predetermined time interval (S503). Thepredetermined time interval is, for example, about 3 minutes, which maybe changed by the user.

If the photographing dates and times are within the predetermined timeinterval (S503: YES), the composition processing unit 1027 confirmswhether or not the background of the search image is the same as thebackground of the display image by pattern matching (S504).

If the backgrounds are the same (S504: YES), the face recognitionprocessing unit 1028 confirms whether or not the person included in thedisplay image is the same as the person included in the search image(S505).

If the same person is not included (S505: NO), the compositionprocessing unit 1027 confirms whether or not the location of the personphotographed in the display image overlaps the location of the personextracted from the search image so as to be composed with the displayimage, within the display image when the search image and the displayimage are superimposed (S506). The determination level may be set insuch a way that the answer is YES if the persons slightly overlap eachother, or YES only if the faces overlap each other.

If the photographing location is the same in both the search image andthe display image (S502: YES), and if the photographing dates and timesare within the predetermined time interval (S503: YES), and if thebackgrounds are the same (S504: YES), and if the same person is notincluded (S505: NO), and if the positions of persons do not overlap(S506: NO), the composition processing unit 1027 extracts the person ofthe search image through contour detection, and attaches the extractedperson to the display image to compose with the display image (S507).Alternatively, the composition processing unit 1027 extracts the personof the display image through contour detection and attaches theextracted person to the search image to compose with the search image.

If the photographing locations of the search image and the display imageare not the same (S502: NO), or if the photographing dates and times arenot within the predetermined time interval (S503: NO), or if thebackgrounds are not the same (S504: NO), or if the same person isincluded (S505: YES), or if the persons are overlapping (S506: YES), thecomposition processing unit 1027 confirms whether or not there isanother image of the same photographing date (S508).

If another image is present (S508: YES), the composition processing unit1027 searches for the next new image (S509) and returns to S502 tocontinue the process. If another image is not present (S508: NO), thecomposition processing unit 1027 ends the process. Note that the orderof S502 and S503 can be changed and these steps correspond to theprocess of searching for a candidate of the second image data.

By performing the above described process from S501 to S509, thecomposition processing unit 1027 searches for an image to be attached tothe display image, extracts the person from the searched image, andcomposes the extracted person.

Further, by providing a geomagnetic sensor in the imaging device,whether or not the orientations of the imaging device are the same, mayalso be added as a condition to the above described determination.

Further, when the imaging device has no GPS, the composition process maybe performed by omitting S502.

FIG. 9 is an illustrative example of the search and composition of imagedata according to the first embodiment.

Image data 901 to 905 are image data that are photographed in series at35° 24′53″ North Latitude and 138° 51′31″ East Longitude from 10:02:00to 10:02:55 on 2015.03.15. The background is the same in the image data901, 902, 904, and 905. The background is different only in the imagedata 903.

The date and time information is obtained by the timer unit 114, and thelocation information is obtained by the GPS receiver 115.

The image data 901 and 902 are image data of the person B photographedby the person A, and the image data 903 is image data of a birdphotographed by the person A or B. The image data 904 and 905 are imagedata of the person A photographed by the person B.

In general, the user repeats photographing operation until satisfactoryimage data is obtained. It is shown that, although both the image data901 and 902 are image data of the person B photographed by the person A,the image data 902 is re-photographed by the person A because the personB of the image data 901 is not smiling. Similarly, it is shown that,although both the image data 904 and 905 are image data of the person Aphotographed by the person B, the image data 905 is re-photographed bythe person B because the person A of the image data 904 is not smiling.Further, the image data 903 shows that the bird suddenly appearingduring a series of photographing operations (representing the period oftime in which the image data 901 to 905 are photographed) isphotographed.

When the shutter button 113-2 of the imaging device 1 is pressed, theimaging device 1 photographs an object and displays the photographedimage on the display unit 112. Then, the imaging device 1 composes theimage when the composition button 113-6 is pressed.

The image data 906 is the image data composed from the image data 902and 905.

The above described FIG. 6A is a display example of the display unit 112just after the photographing of the image data 905 by the imaging device1, or after the reproduction button 113-5 is pressed after thephotographing. Here, when the composition button 113-6 of the imagingdevice 1 is pressed, the imaging device 1 searches for the image data903 according to the flow chart of FIG. 5, and generates the image data906 by compositing the image data 902 and 905. FIG. 6B is a displayexample of the image data 906 composed by the imaging device 1.

The image data 901 to 904 are image data with the same photographinglocation as the photographing location of the image data 905 (S502:YES), and their photographing dates and times are within thepredetermined time interval (S503: YES). The image data 904 includes thesame person as the person of the image data 905 (S505: YES). Thebackground of the image data 903 is different from the image data 905(S504: NO). The image data 901 and 902 have the same background as theimage data 905 (S504: YES), in which the same person is not included(S505: NO), and the persons do not overlap (S506: NO). Of these imagedata, the imaging device 1 selects the image data 902 as the latestimage data, and composes the image data 902 with the image data 905 togenerate the composite image data 906. In the example of FIG. 9, theimage data 905 corresponds to the first image data and the image data902 corresponds to the second image data. The image data 903 is excludedbecause the background is different from the background of the imagedata 905. If the imaging device 1 includes a lens having a zoomfunction, it may be possible to exclude image data with a lens focallength different by a predetermined value or more.

Note that the deviation of view angle of each image data can be absorbedby performing the composition with reference to the background. Forexample, one method is to extract the person B of the image data 902through contour detection to attach the extracted person B to the imagedata 905 with reference to the background. Another possible method is toextract the person A of the image data 905 through contour detection tocompose the extracted person A with the image data 902 with reference tothe background.

FIG. 10 is an illustrative example of the search and composition ofimage data according to the first embodiment.

Image data 1001 and 1002 are image data that are photographed in seriesat 35° 24′53″ North Latitude and 138° 51′31″ East Longitude from10:02:00 to 10:02:50 on 2015.03.16. The background is the same in theimage data 1001 and 1002.

The image data 1001 is the image data of the person B photographed bythe person A. The image data 1002 is the image data of the person Aphotographed by the person B.

When the composition button 113-6 of the imaging device 1 is pressedafter the image data 1002 is photographed, the imaging device 1 composesthe image. Alternatively, the imaging device 1 automatically composesthe image after photographing in the automatic composition mode.

The composite image data 1003 is the image data composed from the imagedata 1001 and 1002.

The image data 1004 is the image of the person B re-photographed by theperson A, after recording the composite image data 1003 in the storageunit 104 or the external recording medium 106 or without recording it.It is shown that while both the image data 1001 and 1004 are the imagedata of the person B photographed by the person A, the image data 1004is re-photographed by the person A because the person B of the imagedata 901 is not smiling. It is also possible that the person Bre-photographs the person A.

The imaging device 1 composes the images again when the compositionbutton 113-6 of the imaging device 1 is pressed after photographing theimage data 1004. Alternatively, the imaging device 1 automaticallycomposes again the image after photographing in the automaticcomposition mode.

The image data 1005 is the image data composed from the image data 1002and 1004.

It is possible to further re-photograph the image to obtain a desiredcomposite image by repeating photographing and composition many times.

As described above, according to the present embodiment, when thecomposition button is pressed after the image to be composed isphotographed, the imaging device can compose the image by searching forthe optimal image. Alternatively, the imaging device can automaticallysearch for the optimal image and compose the image after photographing.

The number of images to be composed is unlimited, and the imaging devicesearches for the optimal image for composition from the photographedimages. Further, a plurality of images to be composed can be treated astargets for composition, as long as the images have the same background,regardless of their photographing timing. Thus, also in the case whenonly one image is re-photographed or when an image that is not a targetfor composition is photographed during the photographing timing of theimage to be composed, it is possible to perform the composition processby eliminating the image not to be composed. Further, it is possible toobtain a desired composite image by repeating the photographing andcomposition operation many times. This makes it possible to increase theoperability for photographing a composite image in such a way that aplurality of persons is present together in one image. Note that whilethe present embodiment has descried the example of selecting a newerimage as a priority, it is also possible to select an image including asmiling person or a full-faced person as a priority.

Second Embodiment

A second embodiment is to perform composition by using the image dataselected by the user.

[Entire Operation of the Imaging Device]

FIG. 11 is a flow chart showing the entire operation of an imagingdevice according to the second embodiment.

Of the steps, S1101 to 51018 are the same as S301 to S308 in FIG. 3, sothat the description thereof will be omitted.

Image data photographed by the imaging unit 110 and stored in the imagetemporary storage area 1032 is displayed on the display unit 112(S1109).

When the reproduction button 113-5 is pressed (S1108: YES) and whenchange of image is selected through the select/determine operationbutton 113-3 (S1111: YES), the imaging device 1 reproduces the imagedata recorded in the storage unit 104 or the external recording medium106 by the image reproduction unit 1025, and displays the reproducedimage data on the display unit 112 (S1110). Here, the select/determineoperation button 113-3 functions as an operation member to select theimage stored in the storage unit 104. Thus, the select/determineoperation button 113-3 corresponds to the selection operation unit.

FIG. 13A is a view showing a display example of the display unit 112when the reproduction button 113-5 is pressed.

When the user selects whether to change the image data to be displayedon the display unit 112 into another image data by the select/determineoperation button 113-3, the image data displayed on the display unit 112is changed to another image data (S1111: YES).

FIG. 13B is a view showing a display example of the display unit 112 inthe imaging device 1 when the display image data is changed to anotherimage data.

Because S1112 is the same as S312 in FIG. 3, the description thereofwill be omitted.

When the composition button 113-6 is pressed (S1112: YES) and whenchange of image is selected through the select/determine operationbutton 113-3 (S1116: YES), the imaging device 1 composes the image datarecorded in the storage unit 104 or the external recording medium 106 bythe composition processing unit 1027 (S1113: YES). The composition isperformed in the image temporary storage area 1032.

Because S1115 is the same as S315 in FIG. 3, the description thereofwill be omitted.

FIG. 14A shows a display example of the display unit 112 when thecomposition button 113-6 is pressed.

The user selects whether to change the image data to be composed intoanother image data through the select/determine operation button 113-3(S1116).

FIG. 14B shows a display example of the display unit 112 in the imagingdevice 1 when the image data to be composed is changed to another imagedata. The imaging device 1 does not change image data 1404 that isselected in S1116, while changing image data 1401 to 1403 to be composedwith the image data 1404.

Because 51117 to S1118 are the same as S317 to S318 in FIG. 3, thedescription thereof will be omitted.

By performing the above described process from S1101 to S1118, theimaging device 1 records, reproduces, and composes the images.

[Image Composition Process]

Next, the image composition process of S1113 will be described. FIG. 12is a flow chart showing an operation example of a composition processingunit according to the second embodiment.

The composition processing unit confirms whether or not there is animage change instruction from the user (S1200).

When there is no image change instruction from the user (S1200: NO), thecomposition processing unit searches for the latest image with the samephotographing date as the image displayed on the display unit (S1201).

Of the steps, S1202 to S1209 are the same as S502 to S509 in FIG. 5, sothat the description thereof will be omitted.

If there is an image change instruction from the user (S1200: YES), orif the photographing locations of the search image and the display imageare not the same (S1202: NO), or if the photographing dates and timesare not within a predetermined time interval (S1203: NO), or if thebackgrounds are not the same (S1204: NO), or if the same person isincluded (S1205: YES), or if the person to be composed overlaps (S1206:YES), the composition processing unit confirms whether or not there isanother image with the same photographing date (S1208). If another imageis present (S1208: YES), the composition processing unit searches forthe next new image (S1209) and returns to S1202, and then continues theprocess. If another image is not present (S1208: NO), the compositionprocessing unit ends the process.

By performing the above described process from S1200 to S1209, thecomposition processing unit searches for an image to be attached to thedisplay image and composes the images.

FIG. 15 is an illustrative example of the search and composition ofimage data according to the second embodiment.

The image data 1501 to 1505 are image data that are photographed inseries at 35° 24′53″ North Latitude and 138° 51′31″ East Longitude from10:02:00 to 10:02:55 on 2015.03.15. The background is the same in theimage data 1501 to 1505. The image data 1501 to 1503 are the image dataof the person B photographed by the person A. The image data 1504 and1505 are the image data of the person A photographed by the person B.

When the shutter button 113-2 of the imaging device 1 is pressed, theimaging device 1 photographs an object, and displays the photographedimage on the display unit 112. Then, the imaging device 1 composes theimage when the composition button 113-6 is pressed.

Image data 1506 is image data that is composed from the image data 1502and 1504.

As described above, according to the present embodiment, it is possibleto obtain the same effect as the first embodiment. In addition, the usercan freely select the image to be composed.

Third Embodiment

A third embodiment is to add only the registered person to the imageafter composition, while deleting the non-registered person from theimage after composition.

[Entire Operation of the Imaging Device]

FIG. 16 is a flow chart showing an entire operation example of theimaging device according to the third embodiment.

Of the steps, S1601 to S1612 are the same as S301 to S312 in FIG. 3, sothat the description thereof will be omitted.

FIG. 18A is a view showing a display example of the display unit 112just after photographing by the imaging device 1 or when thereproduction button 113-5 is pressed. The person A is photographed witha mountain in the background, in which non-registered persons C and Dare also photographed. The persons A and B are registered in advance asimportant persons. Further, when the user photographs a person bytouching the person displayed on the display unit 112 with a touchsensor instead of pressing down the shutter button 113-2, the particularperson may also be registered as an important person.

If the composition button 113-6 is pressed (S1612: YES), the compositionprocessing unit 1027 composes the image data photographed by the imagingunit 110 and stored in the image temporary storage area 1032, with theimage data recorded in the storage unit 104 or the external recordingmedium 106 (S1613). The composition is performed in the image temporarystorage area 1032. At this time, the registered person is added and thenon-registered person is deleted by the composition.

FIG. 18B is a view showing a display example of the image data composedby the imaging device 1. The image including the person A photographedwith a mountain in the background is composed with the person Bphotographed with the same mountain in the background. At the same time,the non-registered persons C and D are deleted.

Because 51615 to S1618 are the same as S315 to S318 in FIG. 3, thedescription thereof will be omitted.

By performing the above described process from S1601 to S1618, theimaging device 1 records, reproduces, and composes the images.

[Image Composition Process]

Next, the image composition process of S1613 will be described.

FIG. 17 is a flow chart showing an operation example of a compositionprocessing unit according to the third embodiment.

Of the steps, S1701 to S1706 are the same as S501 to S506 in FIG. 5, sothat the description thereof will be omitted.

If the photographing locations of the search image and the display imageare the same (S1702: YES), and if the photographing dates and times arewithin a predetermined time interval (S1703: YES), and if thebackgrounds are the same (S1704: YES), and if the same person is notincluded (S1705: NO), and if the persons do not overlap (S1706: NO), thecomposition processing unit 1027 extracts the registered person of thesearch image through contour detection, and composes the extractedperson by attaching to the display image. Alternatively, the compositionprocessing unit 1027 extracts the registered person of the display imagethrough contour detection, and composes the extracted person byattaching to the search image (S1707).

Further, the composition processing unit 1027 deletes the non-registeredperson by extracting the background of the search image at the positionwhere the non-registered person of the display image is present, and byattaching the extracted background to the display image (or composes thebackground data). Alternatively, the composition processing unit 1027deletes the non-registered person by extracting the background of thedisplay image at the position where the non-registered person of thesearch image is present, and by attaching the extracted background tothe search image.

Because S1708 to S1709 are the same as S508 to S509, the descriptionthereof will be omitted.

After the composition of the image (S1707) or if another image with thesame photographing date is not present (S1708: NO), the compositionprocessing unit 1027 confirms whether or not there is a non-registeredperson present in the composed image data (S1710).

If a non-registered person is present in the composed image data (S1710:YES), the composition processing unit 1027 confirms whether or not thereis another image (S1711).

If another image is present (S1711: YES), the composition processingunit 1027 searches and extracts another image (S1712). If another imageis not present (S1711: NO), the composition processing unit 1027 endsthe process.

The composition processing unit 1027 confirms whether or not thephotographing location of the search image and the photographinglocation of the composed image are the same (S1713).

If the photographing locations are the same (S1713: YES), thecomposition processing unit 1027 confirms whether or not thephotographing date and time of the search image and the photographingdate and time of the composed image are within a predetermined timeinterval (S1714). The predetermined time interval is, for example, about3 minutes, which may be changed by the user.

If the photographing dates and times are within the predetermined timeinterval (S1714: YES), the composition processing unit 1027 confirmswhether or not the background of the search image and the background ofthe display image are the same (S1715).

If the backgrounds are the same (S1715: YES), the composition processingunit 1027 deletes the non-registered person (S1716), and returns toS1710.

If the non-registered person is not included (S1710: NO), or if anotherimage of the same photographing date is not present (S1711: NO), thecomposition processing unit 1027 ends the process.

By performing the above described process from S1700 to S1716, thecomposition processing unit 1027 searches for the image to be attachedto the display image, extracts the registered person, and composes theextracted registered person, while deleting the non-registered person.

FIG. 19 is an illustrative example of the search and composition of theimage data.

Image data 1901 to 1903 are image data that are photographed in seriesat 35° 24′53″ North Latitude and 138° 51′31″ East Longitude from10:02:00 to 10:02:50 on 2015.03.16. The background is the same in theimage data 1901 to 1903.

The image data 1901 and 1902 are the image data of the person Bphotographed by the person A, in which the non-registered person D isalso photographed in the image data 1902. It is shown that the imagedata 1902 is re-photographed by the person A because the person B of theimage data 1901 is not smiling. The image data 1903 is the image data ofthe person A photographed by the person B, in which the non-registeredpersons C and D are also photographed. The persons A and B areregistered in advance as important persons. It is also possible thatwhen the user photographs a person by touching the person displayed onthe display unit 112 with a touch sensor instead of pressing down theshutter button 113-2, the particular person may also be registered as animportant person.

When the shutter button 113-2 of the imaging device 1 is pressed, theimaging device 1 photographs an object and displays the photographedimage on the display unit 112. Then, the imaging device 1 composes theimage when the composition button 113-6 is pressed.

The image data 1904 is the image data obtained by compositing theregistered persons from the image data 1902 and 1903, and by deletingthe non-registered person C.

For example, the composition processing unit 1027 extracts the person Bof the image data 1902 through contour detection, and attaches theperson B to the image data 1903 with reference to the background, tocompose the person B with the image data 1902. Further, the compositionprocessing unit 1027 can delete the non-registered person C by attachingthe background of the image data 1902 at the position where thenon-registered person C of the image data 1903 is present, to thenon-registered person C of the image data 1903. It is also possible thatthe composition processing unit 1027 extracts the person A of the imagedata 1903 through contour detection, and composes the person A with theimage data 1902 with reference to the background to generate the imagedata 1904 after composition. If a non-registered person is present ineither one or both of the two image data, the composition processingunit 1027 can extract a registered person from the image data withlarger number of non-registered persons, and to attach the extractedregistered person to the image data with smaller number of registeredpersons.

Next, the composition processing unit 1027 deletes the non-registeredperson D by attaching the background of the image data 1901 at theposition where the non-registered person D of the image data 1904 ispresent, to the non-registered person D of the image data 1904.

The image data 1905 is the image data obtained by deleting thenon-registered person D from the image data 1904 and 1901.

As described above, according to the present embodiment, it is possibleto obtain the same effect as the first embodiment. In addition, it ispossible to add only the registered person to the image and to deletethe non-registered person. Note that while the present embodiment hasdescribed the example of adding the registered person to the image anddeleting the non-registered person, it is also possible to delete thenon-registered person without adding the registered person to the image.

Fourth Embodiment

A fourth embodiment is to search for the background image whenphotographing the image with only the background in which no person isphotographed (hereinafter referred to as the “background image”), andwhen there is no background image, search for image data of photographedbackgrounds publicly available on the network, while deleting anon-registered person by using these backgrounds.

[Hardware Configuration of the Imaging Device]

In FIG. 20A, the same processing units as those in FIG. 1A are denotedby the same reference numerals and the description thereof will beomitted.

FIG. 20A is a block diagram showing a configuration example of theimaging device according to the fourth embodiment. The imaging device 1shown in FIG. 20A includes a wireless communication unit 120. Theimaging device 1 is connected to an external network 3 through an accesspoint device 2 to transmit and receive data with a server device 4 onthe external network 3. It is assumed that the connection to the accesspoint device 2 is established via a wireless connection such as Wi-Fi(registered trademark). The access point device 2 may be a base stationof a mobile network operator. Further, the imaging device 1 may transmitand receive data through a smartphone or mobile terminal, instead ofdirectly transmitting and receiving data with the access point device 2.

The server device 4 includes a plurality of image data with locationinformation as well as a plurality of image data without locationinformation. The imaging device 1 can obtain various image data throughthe access point device 2 and the network 3.

[Software Configuration of the Imaging Device]

FIG. 20B is a software configuration diagram of the imaging device 1 ofthis embodiment, in which the same processing units as those in FIG. 1Bare denoted by the same reference numerals and the description thereofwill be omitted.

FIG. 20B shows a software configuration diagram of the imaging deviceaccording to the fourth embodiment. The ROM 102 includes a wirelessconnection processing unit 1029, to perform the wireless connectionprocess with the access point device 2 by controlling the wirelesscommunication unit 120. Further, the location information acquisitionunit 1022 performs the process of updating the location information ofmetadata of the image file recorded in the storage unit 104 of the bodyor in the external recording medium 106.

[Image Composition Process]

Next, the image composition process will be described. FIG. 21 is a flowchart showing an operation example of the composition processing unitaccording to the fourth embodiment.

Of the steps, S2101 to S2110 are the same as S1701 to S1710 in FIG. 17,so that the description thereof will be omitted.

If a non-registered person is present in the image data (S2110: YES),the composition processing unit 1027 confirms whether or not anotherimage is present in the image storage area 1042 of the storage unit 104(S2111).

If another image is present (S2111: YES), the composition processingunit 1027 searches for another image (S2112). If another image is notpresent (S2111: NO), the composition processing unit 1027 proceeds toS2117.

Because S2113 to S2116 are the same as S1713 to S1716 in FIG. 17, thedescription thereof will be omitted.

If another image is not present in the image storage area 1042 of thestorage unit 104 (S2111: NO), the imaging device 1 connects to theexternal network 3 through the access point device 2 to search for animage of the server device 4 on the external network 3 (S2117). Then,the composition processing unit 1027 performs the same process as S2116by using the image on the network to delete the non-registered person(S2118).

By performing the above described process from S2101 to S2118, thecomposition processing unit 1027 searches for an image to be attached tothe display image, extracts the registered person, and composes theparticular person while deleting the non-registered person.

FIG. 22 is an illustrative example of the search and composition ofimage data according to the fourth embodiment.

Image data 2201 to 2203 are the image data that are photographed inseries at 35° 24′53″ North Latitude and 138° 51′31″ East Longitude from10:02:00 to 10:02:50 on 2015.03.16. The background is the same in theimage data 2201 to 2203.

The image data 2201 is the image data of the person B photographed bythe person A, in which the non-registered person D is also photographed.The image data 2202 is the image data of the person A photographed bythe person B, in which the non-registered persons C and D are alsophotographed. The persons A and B are registered in advance as importantpersons. Further, when the user photographs a person by touching theperson displayed on the display unit 112 with a touch sensor instead ofpressing down the shutter button 113-2, the particular person may alsobe registered as an important person.

When the shutter button 113-2 of the imaging device 1 is pressed, theobject is photographed and the photographed image is displayed on thedisplay unit 112. Then, when the composition button 113-6 is pressed,the image is composed.

The composite image data 2203 is the image data obtained by compositingthe registered persons from the image data 2201 and 2202 while deletingthe non-registered person C.

For example, the composition processing unit 1027 extracts the person Bof the image data 2201 through contour detection, and attaches theextracted person B to the image data 2202 with reference to thebackground, while deleting the non-registered person C by attaching thebackground of the image data 2201 at the position where thenon-registered person C of the image data 2202 is present, to thenon-registered person C of the image data 2202. Alternatively, thecomposition processing unit 1027 extracts the person A of the image data2202 through contour detection, and attaches the extracted person A tothe image data 2201 with reference to the background. If anon-registered person is present in either one or both of the imagedata, the composition processing unit 1027 can extract a registeredperson from the image data with larger number of non-registered personsand attach the extracted registered person to the image data withsmaller number of registered persons.

Then, the non-registered person D can be deleted by re-photographing theimage (background image) with the same background in which thenon-registered person D is not present and by attaching there-photographed image to the non-registered person D.

However, if the non-registered person D does not leave the place for along time, it is difficult to photograph the image (background image)with the same background in which the non-registered person D is notpresent.

Thus, the non-registered person D is deleted by searching for image datapublicly available on the network, and attaching the searched image data2204 (photographed at 35° 24′53″ North Latitude and 138° 51′31″ EastLongitude at the time of 9:30:00 on 2015.03.10). The search of imagedata is performed by pattern matching to find similar image data. It mayalso be possible to search using keywords such as photographinglocation, photographing season, and photographing date and time.

The composite image data 2205 is the image data obtained by deleting thenon-registered person D from the composite image data 2203 by using theimage data 2204 searched on the network.

The composition processing unit 1027 deletes the non-registered person Dby attaching the background of the image data 2204 at the position wherethe non-registered person D of the composite image data 2203 is present,to the non-registered person D of the composite image data 2203.

As described above, according to the present embodiment, it is possibleto obtain the same effect as the third embodiment, and to search forimage data publicly available on the network to delete thenon-registered person by using the searched image data.

Further, it may also be possible to compose the image data photographedby the user in the same location, with reference to the photographinglocation of the image data publicly available on the network.

While the above has described embodiments of the present invention, itgoes without saying that the configurations for implementing thetechnique of the present invention are not limited to the abovedescribed embodiments, and different variations may be possible. Forexample, a part of the configuration of an embodiment can be replaced bythe configuration of another embodiment. Further, the configuration ofanother embodiment can be added to the configuration of an embodiment.All these variations belong to the scope of the present invention.Furthermore, the numbers, messages, and the like that appear in thedescription and drawings are merely examples, and the effects of thepresent invention may not be impaired if different values are used.

Further, the programs described in each of the process examples may beindependent of each other, and a plurality of programs may configure asingle application program. Further, the order of each process may bechanged when performing each of the processes.

With respect to the above described functions or other characteristicsof the present invention, part or all of them may be achieved byhardware, for example, in such a way that they are designed with anintegrated circuit. Further, it may also be possible to achieve them bysoftware in such a way that a microprocessor unit or other devicesinterprets and executes operation programs for achieving the respectivefunctions or other characteristics. The hardware and the software can beused together.

Further, the control lines and information lines shown in the drawingsare those that indicate what is believed necessary for the explanation,and are not intended to represent all the control lines and informationlines required on the product. In fact, almost all configurations can beconsidered as being connected to each other.

LIST OF REFERENCE SINGS

-   1 IMAGING DEVICE-   101 MAIN CONTROLLER-   102 ROM-   103 RAM-   104 STORAGE UNIT-   106 EXTERNAL RECORDING MEDIUM-   110 IMAGING UNIT-   115 GPS RECEIVER-   120 WIRELESS COMMUNICATION UNIT-   1022 LOCATION INFORMATION ACQUISITION UNIT-   1023 IMAGE PROCESSING UNIT-   1024 IMAGE RECORDING UNIT-   1025 IMAGE REPRODUCTION UNIT-   1027 COMPOSITION PROCESSING UNIT-   1028 FACE RECOGNITION PROCESSING UNIT

1. An imaging device comprising: an imager configured to capture animage of an object to generate image data; a memory storing the imagedata; and a processor programmed to perform as: an image reproductionunit for reproducing the image data; a face recognition processing unitfor performing a face recognition process of the image data; acomposition processing unit for generating composite data by performinga composition process so that images of persons captured in each of aplurality of image data are included in one image data; and a selectionoperation unit for allowing a user to select one of the plurality ofimage data stored in the memory; wherein when the image reproductionunit reproduces first image data, the face recognition processing unitrecognizes a face of a first person by performing the face recognitionprocess on the first image data, and wherein in a case where secondimage data, which is obtained by capturing an image of a second personother than the first person at an arbitrary imaging timing differentfrom an imaging timing of the first image data with a backgrounddetermined as the same as a background of the first image data, isstored in the memory, the composition processing unit extracts acandidate of the second image data from the plurality of image datastored in the memory, sets the one of the plurality of image dataselected through the selection operation unit as the second image data,and generates the composite data in which the first person and thesecond person are superimposed on the background determined as the sameby using the first image data and the second image data.
 2. An imagingdevice comprising: an imager configured to capture an image of an objectto generate image data; a memory storing the image data; and a processorprogrammed to perform as: an image reproduction unit for reproducing theimage data; a face recognition processing unit for performing a facerecognition process of the image data; a composition processing unit forgenerating composite data by performing a composition process so thatimages of persons captured in each of a plurality of image data areincluded in one image data; and a mode setting unit for setting anautomatic composition mode in which the composition processing unitperforms the composition process when the image reproduction unitreproduces the image data; wherein the face recognition processing unitrecognizes a face of a first person by performing the face recognitionprocess on first image data, wherein, in a case where second image data,which is obtained by capturing an image of a second person other thanthe first person at an arbitrary imaging timing different from animaging timing of the first image data with a background determined asthe same as a background of the first image data, is stored in thememory, the composition processing unit generates the composite data inwhich the first person and the second person are superimposed on thebackground determined as the same by using the first image data and thesecond image data, and wherein, in a case where the automaticcomposition mode is set, when the image reproduction unit reproduces theimage data, the face recognition processing unit starts the facerecognition process on the image data.
 3. The imaging device accordingto claim 2, wherein, in a case where the image data reproduced by theimage reproduction unit includes an object region, in which an image ofa person is captured, on a position displaced from an image center ofthe image data by a predetermined ratio, the composition processing unitstarts the composition process on the image data.
 4. An imaging devicecomprising: an imager configured to capture an image of an object togenerate image data; a memory storing the image data; a processorprogramed to perform as: a location information acquisition unit foracquiring location information corresponding to a position on which theimage of the object is captured, wherein the memory stores the locationinformation; an image reproduction unit for reproducing the image data;a face recognition processing unit for performing a face recognitionprocess of the image data; and a composition processing unit forgenerating composite data by performing a composition process so thatimages of persons captured in each of a plurality of image data areincluded in one image data; wherein the face recognition processing unitrecognizes a face of a first person by performing the face recognitionprocess on first image data, and wherein, in a case where second imagedata, which is obtained by capturing an image of a second person otherthan the first person at an arbitrary imaging timing different from animaging timing of the first image data with a background determined asthe same as a background of the first image data, is stored in thememory, the composition processing unit extracts a candidate of thesecond image data from the plurality of image data stored in the memorybased on a search condition that image data to be extracted includeslocation information that is the same as location information added tothe first image data, and generates the composite data in which thefirst person and the second person are superimposed on the backgrounddetermined as the same by using the first image data and the secondimage data.
 5. An image processing method comprising the steps of: uponreproduction of image data in which an image of an object is captured,setting an operation mode for starting a composition process using theimage data; generating first image data in which an image of a firstperson is captured; recognizing the first person by performing a facerecognition process on the first image data; searching and reading, froma memory in which a plurality of image data other than the first imagedata is stored, second image data which is obtained by capturing animage of a second person other than the first person with a backgrounddetermined as the same as a background of the first image data; andgenerating composite data in which the first person and the secondperson are superimposed on the background determined as the same.